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Abstract 


Video  cameras  must  produce  images  at  a  reasonable  frame-rate  and  with  a  reasonable  depth  of  field.  These 
requirements  impose  fundamental  physical  limits  on  the  spatial  resolution  of  the  image  detector.  As  a  result, 
current  cameras  produce  videos  with  a  very  low  resolution.  The  resolution  of  videos  can  be  computationally 
enhanced  by  moving  the  camera  and  applying  super-resolution  reconstruction  algorithms.  However,  a  moving 
camera  introduces  motion  blur,  which  limits  super-resolution  quality.  We  analyze  this  effect  and  derive  a 
theoretical  result  showing  that  motion  blur  has  a  substantial  degrading  effect  on  the  performance  of  super 
resolution.  The  conclusion  is,  that  in  order  to  achieve  the  highest  resolution,  motion  blur  should  be  avoided. 

Motion  blur  can  be  minimized  by  sampling  the  space-time  volume  of  the  video  in  a  specific  manner.  We  have 
developed  a  novel  camera,  called  the  ’’jitter  camera,”  that  achieves  this  sampling.  By  applying  an  adaptive 
super-resolution  algorithm  to  the  video  produced  by  the  jitter  camera,  we  show  that  resolution  can  be  notably 
enhanced  for  stationary  or  slowly  moving  objects,  while  it  is  improved  slightly  or  left  unchanged  for  objects 
with  fast  and  complex  motions.  The  end  result  is  a  video  that  has  a  significantly  higher  resolution  than  the 
captured  one. 

Keywords:  Sensors;  Jitter  Camera;  Jitter  Video;  Super  Resolution;  Motion  Blur; 
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Space  Space 


(a)  (b) 

Figure  1:  Conventional  video  cameras  sample  the  continuous  space-time  volume  at  regular  time  inter¬ 
vals  and  fixed  spatial  grid  locations  as  shown  in  (a).  The  space-time  volume  can  be  sampled  differently, 
for  example,  by  varying  the  location  of  the  sampling  grid  as  shown  in  (b)  to  increase  the  resolution  of 
the  video.  A  moving  video  only  approximates  (b)  due  to  motion  blur. 


1.  Why  is  High-Resolution  Video  Hard? 

Improving  the  spatial  resolution  of  a  video  camera  is  different  from  doing  so  with  a  still  camera.  Merely 
increasing  the  number  of  pixels  of  the  detector  reduces  the  amount  of  light  received  by  each  pixel,  and  hence 
increases  the  noise.  With  still  images,  this  can  be  overcome  by  prolonging  the  exposure  time.  In  the  case  of 
video,  however,  the  exposure  time  is  limited  by  the  desired  frame-rate.  The  amount  of  light  incident  on  the 
detector  can  also  be  increased  by  widening  the  aperture,  but  with  a  significant  reduction  of  the  depth  of  field. 
The  spatial  resolution  of  a  video  detector  is  therefore  limited  by  the  noise  level  of  the  detector,  the  frame-rate 
(temporal  resolution)  and  the  required  depth  of  field1.  Our  purpose  is  to  make  a  judicious  use  of  a  given 
detector,  that  will  allow  a  substantial  increase  of  the  video  resolution  by  a  resolution-enhancement  algorithm. 

Figure  1  shows  a  continuous  space-time  video  volume.  A  slice  of  this  volume  at  a  given  time  instance 
corresponds  to  the  image  appearing  on  the  image  plane  of  the  camera  at  this  time.  This  volume  is  sampled 
both  spatially  and  temporally,  where  each  pixel  integrates  light  over  time  and  space.  Conventional  video 
cameras  sample  the  volume  in  a  simple  way,  as  shown  in  Figure  1(a),  with  a  regular  2D  grid  of  pixels 
integrating  over  regular  temporal  intervals  and  at  fixed  spatial  locations.  An  alternative  sampling  of  the  space- 
time  volume  is  shown  in  Figure  1(b).  The  2D  grid  of  pixels  integrates  over  the  same  temporal  intervals,  but 
at  different  spatial  locations.  Given  a  2D  image  detector,  how  should  we  sample  the  space-time  volume  to 

lrThe  optical  transfer  function  of  the  lens  also  imposes  a  limit  on  resolution.  In  this  paper  we  ignore  this  limit  as  it  is  several 
orders  of  magnitudes  above  the  current  resolution  of  video. 
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obtain  the  highest  spatial  resolution2? 

There  is  a  large  body  of  work  on  resolution  enhancement  by  varying  spatial  sampling,  commonly  known 
as  super-resolution  reconstruction  [4, 5,7,9, 13, 17].  Super-resolution  algorithms  typically  assume  that  a  set 
of  displaced  images  are  given  as  input.  With  a  video  camera,  this  can  be  achieved  by  moving  the  camera 
while  capturing  the  video.  However,  the  camera’s  motion  introduces  motion  blur.  This  is  a  key  point  in  this 
paper:  in  order  to  use  super-resolution  with  a  conventional  video  camera,  the  camera  must  move,  but  when 
the  camera  moves,  it  introduces  motion  blur  which  reduces  resolution. 

It  is  well  known  that  an  accurate  estimation  of  the  motion  blur  parameters  is  non-trivial,  and  requires  strong 
assumptions  about  the  camera  motion  during  integration  [2, 13, 15, 19].  In  this  paper,  we  show  that  even  when 
an  accurate  estimate  of  the  motion  blur  parameters  is  available,  motion  blur  has  a  significant  influence  on 
the  super-resolution  result.  We  derive  a  theoretical  lower  bound,  indicating  that  the  expected  performance  of 
any  super-resolution  reconstruction  algorithm  deteriorates  as  a  function  of  the  motion  blur  magnitude.  The 
conclusion  is  that,  in  order  to  achieve  the  highest  resolution,  motion  blur  should  be  avoided. 

To  achieve  this,  we  propose  the  “jitter  camera,”  a  novel  video  camera  that  samples  the  space-time  volume  at 
different  locations  without  introducing  motion  blur.  This  is  done  by  instantaneously  shifting  the  detector  (e.g. 
CCD)  between  temporal  integration  periods,  rather  than  continuously  moving  the  entire  video  camera  during 
the  integration  periods.  We  have  built  a  jitter  camera,  and  developed  an  adaptive  super-resolution  algorithm  to 
handle  complex  scenes  containing  multiple  moving  objects.  By  applying  the  algorithm  to  the  video  produced 
by  the  jitter  camera,  we  show  that  resolution  can  be  enhanced  significantly  for  stationary  or  slowly  moving 
objects,  while  it  is  improved  slightly  or  left  unchanged  for  objects  with  fast  and  complex  motions.  The  end 
result  is  a  video  that  has  higher  resolution  than  the  captured  one. 

2.  How  Bad  is  Motion  Blur  for  Super-resolution? 

The  influence  of  motion  blur  on  super  resolution  is  well  understood  when  all  input  images  undergo  the  same 
motion  blur  [1, 10].  It  becomes  more  complex  when  the  input  images  undergo  different  motion  blurs,  and 
details  that  appear  blurred  in  one  image,  appear  sharp  in  another  image.  We  address  the  influence  of  motion 
blur  for  any  combination  of  blur  orientations. 

increasing  the  temporal  resolution  [18]  is  not  addressed  in  this  paper. 
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Figure  2:  We  measure  the  super-resolution  “hardness”  by  the  volume  of  plausible  high-resolution  so¬ 
lutions  [1].  The  volume  of  solutions  is  proportional  to  s(A)n 2  where  n2  is  the  high  resolution  image  size. 


The  graphs  show  the  value  of  s(A)  as  a  function  of  the  length  of  the  motion-blur  trajectories  { ||(j  ||  }^=0. 
We  show  a  large  number  of  graphs  computed  for  different  configurations  of  blur  orientations.  The 
thick  graph  (blue  line)  is  the  lower  bound  of  s(A),  for  any  combination  of  motion  blur  orientations.  In 
all  shown  configurations,  the  motion  blur  has  a  significant  influence  on  s(A)  and  hence  on  the  volume 
of  solutions.  The  increase  in  the  volume  of  solutions  can  explain  the  increase  in  reconstruction  error  in 
super-resolution  shown  in  Figure  3. 


Super-resolution  algorithms  estimate  the  high  resolution  image  by  modeling  and  inverting  the  imaging  pro¬ 
cess.  Analyzing  the  influence  of  motion  blur  requires  a  definition  for  super-resolution  “hardness”  or  the 
“invertibility”  of  the  imaging  process.  We  use  a  linear  model  for  the  imaging  process  [1,7,9, 13],  where  the 
intensity  of  a  pixel  in  the  input  image  is  presented  as  a  linear  combination  of  the  intensities  in  the  unknown 
high  resolution  image: 


y  =  Ax  +  z, 


where  x  is  a  vectorization  of  the  unknown  discrete  high  resolution  image,  y  is  a  vectorization  of  all  the  input 
images,  and  the  imaging  matrix  A  encapsulates  the  camera  displacements,  blur  and  decimation  [7],  The 
random  variable  z  represents  the  uncertainty  in  the  measurements  due  to  noise,  quantization  error  and  model 


inaccuracies. 
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Baker  and  Kanade  [1]  addressed  the  invertibility  of  the  imaging  process  in  a  noise-free  scenario,  where  £ 
represents  the  quantization  error.  In  this  case,  each  quantized  input  pixel  defines  two  inequality  constraints 
on  the  super-resolution  solution.  The  combination  of  constraints  forms  a  volume  of  solutions  that  satisfy  all 
quantization  constraints.  Baker  and  Kanade  suggest  to  use  the  volume  of  solutions  as  a  measure  of  uncertainty 
in  the  super  resolution  solution.  Their  paper  [1]  shows  the  benefits  in  measuring  the  volume  of  solutions  over 
the  standard  matrix  conditioning  analysis. 


We  measure  the  influence  of  motion  blur  by  the  volume  of  solutions.  To  keep  the  analysis  simple,  the 
following  assumptions  are  made.  First,  the  motion  blur  in  each  input  image  is  induced  by  a  constant  velocity 
motion.  Different  input  images  may  have  different  motion  blur  orientations.  Second,  the  optical  blur  is  shift- 
invariant.  Third,  the  input  images  are  related  geometrically  by  a  2D  translation.  Fourth,  the  number  of  input 
pixels  equals  the  number  of  output  pixels.  Under  the  last  assumption,  the  dimensionality  n 2  of  x  equals  the 
dimensionality  of  y.  Since  the  uncertainty  due  to  quantization  is  an  n 2  -dimensional  unit  cube,  the  volume  of 
solutions  for  a  given  imaging  matrix  A  can  be  computed  from  the  absolute  value  of  its  determinant 


vol(A) 


1 

Rf 


(2) 


In  Appendix  A,  we  derive  a  simplified  expression  for  |  A\  as  a  function  of  the  imaging  parameters.  This  allows 
for  an  efficient  computation  of  vol(A),  as  well  as  a  derivation  of  a  lower  bound  on  vol(A  ')  as  a  function  of  the 
extent  of  motion  blur. 

Since  the  volume  of  solutions  vol(A)  depends  on  the  image  size  which  is  n2,  we  define  in  Appendix  A 
(equation  8)  a  function  s(A)  such  that: 

vol(A)  oc  s(A)n2 . 

s(A)  has  two  desirable  properties  for  analyzing  the  influence  of  motion  blur.  First,  it  is  independent  of  the 
camera’s  optical  transfer  function  and  the  detector’s  integration  function,  and  normalized  to  one  when  there 
is  no  motion  blur  and  the  camera  displacements  are  optimal  (Appendix  B).  Second,  vol(A)  is  exponential  in 
the  image  size  whereas  s(A)  is  normalized  to  account  for  the  image  size. 

Figure  2  shows  s(A)  as  a  function  of  the  lengths  of  the  motion  blur  trajectories.  Specifically,  let  f  be  a  vector 
describing  the  motion  blur  trajectory  for  the  j-th  input  image:  During  integration,  the  projected  image  moves 
at  a  constant  velocity  from  —  to  lf.  Each  graph  in  figure  2  shows  the  value  of  s(A)  as  a  function  of  the  length 
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Figure  3:  The  effect  of  motion  blur  on  super-resolution  with  a  known  simulated  motion  blur,  (a)  The 
top  image  is  the  original  ground-truth  image.  The  middle  image  is  the  super-resolution  result  for  4 
simulated  input  images  with  no  motion  blur.  This  image  is  almost  identical  to  the  ground  truth  image. 
The  bottom  image  is  a  super-resolution  result  for  4  simulated  input  images  with  motion  blur  of  3.5 
pixels.  Two  images  with  horizontal  blur  and  two  with  vertical  blur  were  used.  The  algorithm  used  the 
known  simulated  motion  blur  kernels  and  the  known  displacements.  The  degradation  in  the  super¬ 
resolution  result  due  to  motion  blur  is  clearly  visible,  (b)  The  graph  shows  the  grey  level  RMS  error  in 
the  super-resolution  image  as  a  function  of  motion  blur  trajectory  length. 


(a)  (b) 

Figure  4:  A  jitter  video  camera  shifts  the  sampling  grid  accurately  and  instantaneously.  This  can  be 
achieved  using  micro-actuators,  which  are  both  fast  and  accurate.  The  actuator  can  shift  the  detector 
as  shown  in  (a),  or  it  can  be  used  to  operate  a  simple  optical  device,  such  as  the  tilted  glass  plate  shown 
in  (b),  in  order  to  optically  move  the  image  with  respect  to  the  static  detector. 

of  the  four  motion  blur  trajectories  {||^j|}^=0.  The  different  graphs  correspond  to  different  configurations  of 
blur  orientations  in  four  input  images.  The  graphs  were  computed  for  optimal  camera  displacements  (see 
Appendix  B)  and  magnification  factor  2. 

It  can  be  seen  that  in  all  selected  motion  blur  configurations  s(A)  cx  vol  (A)  ^  increases  as  a  function  of 
the  length  of  the  motion  blur  trajectories  {||Zjj|}.  The  thick  blue  line  is  the  lower  bound  of  s(A),  whose 
derivation  can  be  found  in  Appendix  A.  This  bound  is  for  any  configuration  of  blur  orientations  and  any 
camera  displacements. 

The  findings  above  confirm  that,  at  least  for  our  assumptions,  any  motion  blur  is  bad  for  super-resolution  and 
the  larger  the  motion  blur,  the  larger  the  volume  of  solutions. 

Figure  3(a)  shows  super-resolution  results  of  simulations  with  and  without  motion  blur.  Motion  blur  as  small 
as  3.5  pixels  degrades  the  super-resolution  result,  such  that  some  of  the  letters  are  unreadable.  Figure  3(b) 
presents  the  RMS  error  in  the  reconstructed  super-resolution  image  as  a  function  of  the  extent  of  the  motion 
blur.  It  can  be  seen  that  the  RMS  error  increases  as  a  function  of  the  motion  blur  magnitude.  This  effect  is 
consistent  with  the  theoretical  observations  made  above. 

3.  Jitter  Video:  Sampling  without  Motion  Blur 

Our  analysis  showed  that  sampling  with  minimal  motion  blur  is  important  for  super-resolution.  Little  can  be 
done  to  prevent  motion  blur  when  the  camera  is  moving3  or  when  objects  in  the  scene  are  moving.  Therefore, 

3  Small  camera  shakes  can  be  eliminated  by  optical  lens  stabilization  systems,  which  stabilize  the  image  before  it  is  integrated. 
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our  main  goal  is  to  sample  at  different  spatial  locations  while  avoiding  motion  blur  in  static  regions  of  the 
image. 

The  key  to  avoiding  motion  blur  is  synchronous  and  instantaneous  shifts  of  the  sampling  grid  between  tem¬ 
poral  integration  periods,  rather  than  a  continuous  motion  during  the  integration  periods.  In  Appendix  B  we 
show  that  the  volume  of  solutions  can  be  minimized  by  properly  selecting  the  grid  displacements.  For  exam¬ 
ple,  in  the  case  of  four  input  images,  one  set  of  optimal  displacements  is  achieved  by  shifting  the  sampling 
grid  by  half  a  pixel  horizontally  and  vertically.  Implementing  these  abrupt  shifts  by  moving  a  standard  video 
camera  with  a  variable  magnification  factor  is  non-trivial4.  Hence  we  propose  to  implement  the  shifts  of  the 
sampling  grid  inside  the  camera. 

Figure  4  shows  two  possible  ways  to  shift  the  sampling  grid  instantaneously.  Figure  4(a)  shows  a  purely 
mechanical  design,  where  the  detector  (e.g.  CCD)  is  shifted  by  actuators  to  change  the  sampling  grid  location. 
If  the  actuators  are  fast  and  are  activated  synchronously  with  the  reading  cycle  of  the  detector,  then  the 
acquired  image  will  have  no  motion  blur  due  to  the  shift  of  the  detector.  Figure  4(b)  shows  a  mechanical- 
optical  design.  A  flat  thin  glass  plate  is  used  to  shift  the  image  over  the  detector.  An  angular  change  of 
a  1mm  thick  plate  by  one  degree  shifts  the  image  by  5.8 /im,  which  is  of  the  order  of  a  pixel  size.  Since 
the  displacement  is  very  small  relative  to  the  focal  length,  the  change  of  the  optical  path  length  results  with 
negligible  effect  on  the  focus  (the  point  spread  area  is  much  smaller  than  the  area  of  a  pixel).  The 
mechanical-optical  design  shown  Figure  4(b)  has  been  used  for  high-resolution  still-imaging,  for  example  by 
Pixera  [6],  where  video  related  issues  such  as  motion  blur  and  dynamic  scenes  do  not  arise. 

An  important  point  to  consider  in  the  design  of  a  jitter  camera  is  the  quality  of  the  camera  lens.  With  standard 

video  cameras,  the  lens-detector  pair  is  matched  to  reduce  spatial  aliasing  in  the  detector.  For  a  given  detector, 

the  matching  lens  attenuates  the  spatial  frequencies  higher  than  the  Nyquist  frequency  of  the  detector.  For  a 

jitter  camera,  higher  frequencies  are  useful  since  they  are  exploited  in  the  extraction  of  the  high  resolution 

video.  Hence,  the  selected  lens  should  match  a  detector  with  a  higher  (the  desired)  spatial  resolution. 

4  A  small  uniform  image  displacement  can  be  approximated  by  rotating  the  camera  about  the  X,  Y  axes.  However,  the  rotation 

extent  depends  on  the  exact  magnification  factor  of  the  camera,  which  is  hard  to  obtain.  In  addition,  due  to  camera’s  mass,  abrupt 
shifting  of  the  camera  is  challenging. 
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4.  The  Jitter  Camera  Prototype 

To  test  our  approach,  we  have  built  the  jitter  camera  prototype  shown  in  Figure  5.  This  camera  was  built  using 
a  standard  16mm  television  lens,  a  Point-Grey  [16]  Dragon-Fly  board  camera,  and  two  Physik  Instrumente  [8] 
micro-actuators.  The  micro-actuators  and  the  board  camera  were  controlled  and  synchronized  by  a  Physik 
Instrumente  Mercury  stand-alone  controllers  (not  shown). 

The  jitter  camera  is  connected  to  a  computer  using  a  standard  firewire  interface,  and  therefore  it  appears  to 
be  a  regular  firewire  camera. 

We  used  in  our  prototype  two  DC-motor  actuators,  which  enable  a  frame-rate  of  approximately  8  frames  per 
second.  Newly  developed  piezoelectric  based  actuators  can  offer  much  higher  speed  than  DC-motor  based 
actuators.  Such  actuators  are  already  used  for  camera  shake  compensation  by  Minolta  [12],  however  they  are 
less  convenient  for  prototyping  at  this  point  in  time. 

The  camera  operates  as  follows: 

1 .  At  power  up  the  actuators  are  moved  to  a  fixed  home-position. 

2.  For  each  sampling  position  in  [(0,0), (0,0.5), (0.5, 0.5),  (0.5,0]  pixels  do 

•  Move  the  actuators  to  the  next  sampling  position. 

•  Bring  the  actuators  to  a  full  stop. 

•  Send  a  trigger  signal  to  the  camera  to  initiate  frame  integration  and  wait  during  integration  dura¬ 
tion. 

•  When  the  frame  is  ready,  the  camera  sends  it  to  the  computer  over  the  Firewire  interface. 

3.  End  loop 

4.  Repeat  process  from  step  (2). 

To  evaluate  the  accuracy  of  the  jitter  mechanism  we  captured  a  sequence  of  images  with  the  jitter  camera, 
computed  the  motion  between  frames  to  a  sub-pixel  accuracy  [3]  and  compared  the  computed  motion  to  the 
expected  value.  The  results  are  shown  in  Figure  6.  The  green  circles  show  the  expected  displacements,  and 
the  red  diamonds  show  the  actual  displacements  over  multiple  cycles.  We  can  see  that  the  accuracy  of  the 
jitter  mechanism  was  better  than  0.1  pixel.  We  can  also  see  that  while  some  error  is  accumulated  along  the 
path,  the  camera  accurately  returns  to  its  zero  position,  thus  preventing  drift. 
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Board  Camera 


Figure  5:  The  jitter  camera  prototype  shown  with  its  cover  open.  The  mechanical  micro-actuators  are 
used  for  shifting  the  board  camera.  The  two  actuators  and  the  board  camera  are  synchronized  such 
that  the  camera  is  motionless  during  integration  time. 


The  resolution  of  the  computed  high-resolution  video  was  1280  x  960,  which  has  four  times  the  number  of 
pixels  compared  to  the  resolution  of  the  input  video,  which  was  640  x  480.  This  enhancement  upgrades  an 
NTSC  grade  camera  to  an  HTDV  grade  camera  while  maintaining  the  depth  of  field  and  the  frame-rate  of 
the  original  camera. 

With  the  recent  advances  in  micro-electric  mechanical  systems  (MEMS),  it  will  hopefully  be  possible  to 
embed  the  jitter  mechanism  within  the  detector  chip,  thus  creating  a  jitter-detector. 

5.  Adaptive  Super-resolution  for  Dynamic  Scenes 

Given  a  video  sequence  captured  by  a  jitter  camera,  we  would  like  to  compute  a  high  resolution  video  using 
super-resolution.  We  have  chosen  iterated-back-projection  [9]  as  the  super-resolution  algorithm.  Iterated- 
back-projection  was  shown  in  [4]  to  produce  high  quality  results  and  is  simple  to  implement  for  videos 
containing  complex  scenes.  The  main  challenge  in  our  implementation  is  handling  multiple  motions  and 
occlusions.  Failing  to  cope  with  these  problems  results  in  strong  artifacts  that  render  the  output  useless. 

To  address  these  problems,  we  compute  the  image  motion  in  small  blocks,  and  detect  blocks  suspected  of 
having  multiple  motions.  The  adaptive  super-resolution  algorithm  maximizes  the  use  of  the  available  data  for 
each  block. 
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O  Expected  Position.  ♦  Actual  Position. 

Figure  6:  Accuracy  of  the  jitter  mechanism.  The  detector  moves  one  step  at  a  time  along  the  path  shown 
by  the  blue  arrows.  The  green  circles  show  the  expected  position  of  exactly  half  a  pixel  displacement 
and  the  red  diamonds  show  the  actual  position  over  multiple  cycles.  We  can  see  that  the  accuracy  was 
less  than  tenth  of  a  pixel.  We  can  also  see  that  he  jitter  mechanism  returns  very  accurately  to  its  zero 
position,  hence  prevents  excessive  error  accumulation  over  multiple  cycles. 

5.1  Motion  Estimation  in  the  Presence  of  Aliasing 

The  estimation  of  image  motion  should  be  robust  to  outliers,  which  are  mainly  caused  by  occlusions  and 
multiple  motions  within  a  block.  To  address  this  problem,  we  use  the  Tukey  M-estimator  error  function  [11]. 
The  Tukey  M-estimator  depends  on  a  scale  parameter  a,  the  standard  deviation  of  the  gray-scale  differences 
of  correctly-aligned  image  regions  ( inlier  regions ). 

Due  to  the  under-sampling  of  the  image,  gray-scale  image  differences  in  the  inlier  regions  are  dominated  by 
aliasing,  and  are  especially  significant  near  sharp  image  edges.  Hence  we  approximate  the  standard  deviation 
of  the  gray-scale  differences  a  in  each  block  from  the  standard  deviation  of  the  aliasing  aa  in  the  block,  as 
a  =  \f2ua.  This  approximation  neglects  the  influence  of  noise,  and  makes  the  simplifying  assumption  that 
the  aliasing  effects  in  two  aligned  blocks  are  statistically  uncorrelated.  In  the  following  we  describe  the 
approximation  for  the  standard  deviation  of  the  aliasing  in  each  block  oa,  using  results  on  the  statistics  of 
natural  images. 

Let  /  be  a  high  resolution  image,  blurred  and  decimated  to  obtain  a  low  resolution  image  g: 

9  =  (f*h)  | 

where  *  denotes  convolution  and  j  denotes  subsampling.  Let  s  be  a  perfect  rect  low  pass  filter.  The  aliasing 
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in  g  is  given  by: 


(f*h  —  f*s*h)  l—  f*h*  (8  —  s)  l 

The  band-pass  filter  h*  (8  —  s)  can  hence  be  used  to  simulate  aliasing.  For  the  motion  estimation,  we  need 
to  estimate  aa,  the  standard  deviation  of  the  response  of  this  filter  to  blocks  of  the  unknown  high  resolution 
image.  We  use  the  response  of  this  filter  to  the  aliased  low  resolution  input  images  to  estimate  aa.  Let  cr0 
be  the  standard  deviation  of  the  filter  response  to  an  input  block.  Testing  with  a  large  number  of  images,  we 
found  that  aa  can  be  approximated  to  be  a  linear  function  of  <r0.  Similar  results  for  non- aliased  images  were 
shown  by  Simoncelli  [20]  for  various  band-pass  filters  at  different  scales.  For  blocks  of  size  16  x  16  pixels 
the  linear  coefficient  was  in  the  range  [0.5, 0.7].  In  the  experiments,  we  set  oa  =  0.7 a0  which  was  sufficient 
for  our  purpose. 

5.2  Adaptive  Data  Selection 

We  use  the  scale  estimate  a  from  the  previous  section  to  differentiate  between  blocks  with  a  single  motion 
and  blocks  that  may  have  multiple  motions  and  occlusions.  A  block  in  which  the  SSD  error  exceeds  3a 
is  excluded  from  the  super-resolution  calculation.  In  order  to  double  the  resolution  (both  horizontally  and 
vertically)  three  additional  valid  blocks  are  needed  for  each  block  in  the  current  frame.  Depending  on  the 
timing  of  the  occlusions,  these  additional  blocks  could  be  found  in  previous  frames  only,  in  successive  frames 
only,  both,  or  not  at  all.  We  therefore  search  for  valid  blocks  in  both  temporal  directions  and  select  the  blocks 
which  are  valid  and  closest  in  time  to  the  current  frame. 

In  blocks  containing  a  complex  motion,  it  may  happen  that  less  than  four  valid  blocks  are  found  within  the 
temporal  search  window.  In  this  case,  although  the  super-resolution  image  is  under-constrained,  iterated- 
back-projection  produces  reasonable  results  [4],  Figure  7  shows  an  example  from  an  outdoor  video  sequence 
containing  multiple  moving  objects.  On  bottom  is  a  visualization  of  the  number  of  valid  blocks  used  for  each 
block  in  this  frame.  Blocks  where  less  than  four  valid  blocks  were  used  are  darkened. 

6.  Experiments 

We  tested  resolution  enhancement  with  our  jitter  camera  for  both  static  and  dynamic  scenes.  The  input  images 
were  obtained  from  the  raw  Bayer-pattem  samples  using  the  de-mosaicing  algorithm  provided  by  the  camera 
manufacturer  [16].  The  images  were  then  transformed  to  the  CIE-Lab  color  space,  and  the  super  resolution 
algorithm  [9]  was  applied  to  the  L-channel  only.  The  low  resolution  (a,b)-chroma  channels  were  linearly 
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Blocks  usage  map 

Figure  7:  Adaptation  of  the  super-resolution  algorithm  to  moving  objects  and  occlusions.  The  image  on 
top  shows  one  frame  from  a  video  sequence  of  a  dynamic  scene.  The  image  on  bottom  is  a  visualization 
of  the  number  of  valid  blocks,  from  four  frames,  used  by  the  algorithm  in  each  block.  We  darkened 
blocks  where  the  algorithm  used  less  than  four  valid  blocks  due  to  occlusions. 


interpolated  and  combined  with  the  high  resolution  L-channel. 

6.1  Resolution  Tests 

The  resolution  enhancement  was  evaluated  quantitatively  using  a  standard  Kodak  test  target.  The  input  to 
the  super-resolution  algorithm  was  four  frames  from  a  jitter-camera  video  sequence.  Figure  8  shows  angular, 
vertical  and  horizontal  test  patterns,  the  aliasing  effects  are  clearly  seen  in  the  input  images,  where  the  line 
separation  is  not  clear  even  at  the  lower  resolution  of  60  lines  per  inch.  In  the  computed  super-resolution 
images  the  spatial  resolution  is  clearly  enhanced  in  all  angles  and  it  is  possible  to  resolve  separate  lines  well 
above  100  lines  per  inch. 
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Raw  video  from  jitter  camera  Super-resolution  output 


Figure  8:  Resolution  test  using  a  standard  Kodak  test  target.  The  left  column  shows  angular,  vertical 
and  horizontal  resolution  test  targets  that  were  captured  by  the  jitter  camera  (one  of  four  input  images). 
The  right  column  shows  the  super-resolution  results.  Note  the  strong  aliasing  in  the  input  images  and 
the  clear  separation  between  lines  in  the  super-resolution  result  images. 
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6.2  Color  Test 


The  standard  Kodak  test  target  is  black  and  white.  In  order  to  check  the  color  performance,  we  used  a  test 
target  consisting  of  a  color  image  and  lines  of  text  of  different  font  sizes.  Figures  9(a), (b)  show  one  out  of 
four  different  input  images  taken  by  the  jitter  camera  and  a  magnified  part  of  the  image.  For  the  input  images 
we  utilized  the  best  color  de-mosaicing  algorithm  the  Dragonfly  camera  had  to  offer  (proprietary  ’rigorous’ 
algorithm).  We  can  see  that  the  input  image  contains  color  artifact  along  edges.  Figures  9(c), (d)  show  the 
super-resolution  result  image  and  a  magnified  part  respectively.  The  resolution  is  clearly  enhanced  and  it  is 
now  possible  to  read  all  the  text  lines  that  were  unreadable  in  the  input  images.  Moreover,  we  can  see  that 
the  de-mosaicing  artifacts  have  almost  completely  disappeared,  while  the  colors  were  preserved.  This  is  due 
to  the  fact  that  the  super  resolution  was  applied  only  to  the  intensity  channel  while  the  chromaticity  channels 
were  smoothly  interpolated. 

6.3  Dynamic  Video  Tests 

Several  experiments  were  conducted  to  test  the  system’s  performance  in  the  presence  of  moving  objects  and 
occlusions.  Figure  10  shows  magnified  parts  of  a  scene  with  mostly  static  objects.  These  objects,  such  as 
the  crossing  pedestrians  sign  in  the  first  row  and  the  no-parking  sign  in  the  second  row  were  significantly 
enhanced,  revealing  new  details.  Figure  1 1  shows  magnified  parts  of  scenes  with  static  and  dynamic  objects. 
One  can  see  that  the  adaptive  super-resolution  algorithm  has  increased  the  resolution  of  stationary  objects 
while  preserving  or  increasing  the  resolution  of  moving  objects. 

7.  Conclusions 

Super-resolution  algorithms  can  improve  spatial  resolution.  However,  their  performance  depends  on  various 
factors  in  the  camera  imaging  process.  We  showed  that  motion  blur  causes  significant  degradation  of  super¬ 
resolution  results,  even  when  the  motion  blur  function  is  known.  The  proposed  solution  is  the  jitter  camera, 
a  video  camera  capable  of  sampling  the  space-time  volume  without  introducing  motion  blur.  Applying  a 
super-resolution  algorithm  to  jitter  camera  video  sequences  significantly  enhances  their  resolution. 

Image  detectors  are  becoming  smaller  and  lighter  and  thus  require  very  little  force  to  jitter.  With  recent 
advances  it  may  be  possible  to  manufacture  jitter  cameras  with  the  jitter  mechanism  embedded  inside  the 
detector  chip.  Jittering  can  then  be  added  to  regular  video  cameras  as  an  option  that  enables  a  significant 
increase  of  spatial  resolution  while  keeping  other  factors  such  as  frame-rate  unchanged. 
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Figure  9:  Resolution  test  of  combined  color  and  text  image.  Panels  (a),(b)  show  one  out  of  four  different 
input  images  taken  by  the  jitter  camera  together  with  a  magnified  part  of  the  image.  Note  that  the  last 
line  of  the  text,  which  is  only  six  pixels  high,  is  completely  unreadable;  also,  note  the  de-mosaicing 
artifacts  in  both  the  text  and  the  image.  Panels  (c),(d)  show  the  super-resolution  result  and  a  magnified 
part  of  it.  The  resolution  is  clearly  enhanced  and  it  is  now  possible  to  read  all  the  text  lines  that  were 
unreadable  in  the  input  images.  Moreover,  we  can  see  that  the  de-mosaicing  artifacts  have  almost 
vanished  while  the  colors  were  preserved. 


Motion  blur  is  only  one  factor  in  the  imaging  process.  By  considering  other  factors,  novel  methods  for 
sampling  the  space-time  volume  can  be  developed,  resulting  in  further  improvements  in  video  resolution.  In 
this  paper,  for  example,  we  limited  the  detector  to  a  regular  sampling  lattice  and  to  regular  temporal  sampling. 
One  interesting  direction  can  be  the  use  of  different  lattices  and  different  temporal  samplings.  We  therefore 
consider  the  jitter  camera  to  be  a  first  step  towards  a  family  of  novel  camera  designs  that  better  sample  the 
space-time  volume  to  improve  not  only  spatial  resolution,  but  also  temporal  resolution  and  spectral  resolution. 
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Figure  10:  Jitter  camera  super-resolution  for  scenes  of  mostly  stationary  objects.  The  left  column 
shows  the  raw  video  input  from  the  jitter  camera  and  the  right  column  shows  the  super-resolution 
results.  The  first  row  shows  a  static  scene.  Note  the  significant  resolution  enhancement  of  the  pedestrian 
on  the  sign,  and  the  fine  texture  of  the  tree  branches.  The  second  row  shows  a  scene  with  few  moving 
objects.  Note  the  enhancement  of  the  text  on  the  no-parking  sign,  and  some  enhancement  of  the  walking 
person. 
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Raw  video  from  jitter  camera 


Super-resolution  output 
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Figure  11:  Jitter  camera  super-resolution  for  scenes  with  dynamic  and  stationary  objects.  The  left  col¬ 
umn  shows  the  raw  video  input  from  the  jitter  camera  and  the  right  column  shows  the  super-resolution 
results.  The  first  row  shows  a  scene  with  a  large  stationary  object  (boat)  and  a  large  moving  object 
(woman’s  head).  As  expected,  the  resolution  enhancement  is  better  for  the  boat.  The  second  row  shows 
a  particularly  dynamic  scene  with  many  moving  objects.  Note  the  enhancement  of  the  face  of  the 
walking  women  (center)  and  the  kid  on  the  scooter(left). 
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Appendix  A.  The  Influence  of  Motion  Blur  on  the  Volume  of  Solutions 

The  imaging  process  of  the  multiple  input  images  is  modelled  by  a  matrix  A: 

y  =  Ax  +  z.  (3) 


x  is  a  vectorization  of  the  unknown  discrete  high  resolution  image,  y  is  a  vectorization  of  all  the  input 
images,  and  z  is  the  uncertainty  in  measurements.  A  minimal  number  of  input  images  is  assumed,  such  that 
the  dimensionality  of  y  equals  to  the  dimensionality  of  x,  and  matrix  A  is  square. 

The  volume  of  solutions  corresponding  to  a  square  imaging  matrix  A  is  computed  from  the  absolute  value  of 
its  determinant  (equation  2): 

vol(A)  = 

In  the  following  we  derive  a  simplified  expression  for  the  determinant  of  the  imaging  matrix  A,  and  present 
the  volume  of  solutions  as  a  function  of  the  camera  displacements,  motion  blurs,  optical  transfer  function, 
and  the  integration  function  of  the  detector. 

Let  f  be  the  n  x  n  high  resolution  image  (corresponding  to  x  in  equation  3),  and  let  {g,}"^1  be  the  ^  x  ^ 
input  images  (corresponding  to  y).  The  imaging  process  is  defined  in  the  image  domain  by: 


1 

w 


gj  =  (f  *  h/)  !„,  -|  Zj,  (4) 

where  *  denotes  convolution,  h j  encapsulates  the  sensor  displacement  and  motion  blur  of  the  j-th  image  and 
the  optical  blur  and  detector  integration  of  the  camera,  z  j  represents  the  quantization  error,  and  [m  denotes 
subsampling  by  a  factor  of  m.  In  the  frequency  domain,  let  Zj,  Gj,  Hj ,  F  denote  the  Fourier  transforms  of 
Zj,  gj,hj,  f,  respectively.  The  frequencies  of  the  high  resolution  image  are  folded  as  a  result  of  the  subsam¬ 
pling: 

Gj(u,v)  —  Zj(u,v)+  ^2  Rect[_^^](u,v)Hj(u,v)F(u,v),  (5) 

neu.vev 

where  U={u  +  V={v  +  !^}kL_00,  and  Rect{_^^{u,v)  equals  1  when  [-f  <  u,  v  <  f]  and  0  otherwise. 

This  leads  to  the  following  result. 

Proposition  1  Let  A  be  the  matrix  of  equation  3,  corresponding  to  the  imaging  process  above  ( Equation  4) 
for  m  =  2  (four  input  images). 
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Defineu  =  u  —  sign(u)^  ,  v  =  v  —  sign(v)  then  the  determinant  of  A  is  given  by :  |A|  =  \\_n<uv<n  |AU)„|, 
where: 

H0(u,v)  H0(u,v)  H0(u,v)  H0(u,v) 

Hf  U,  V  )  U,  V  )  H,(  U,  V  )  # i  (u,  v) 

AU,V 

H2(u,v)  H2(u,v )  H2(u,v)  H2(u,v) 

H3(u,v)  H3(u,v)  H3(u,v)  H3(u,v) 

Proof:  Let  Abe  a  matrix  describing  the  imaging  process  in  the  frequency  domain 


- 1 

O 

T 

I 

eicN 

1 

feT 

1 - 

99 

i _ 

=  A 

+ 

_  Gj(2  - 1,5  - 1)  _ 

_f(i- 1, 1  - 1))  _ 

_z3(i-  i,5-i). 

From  equation  5,  in  the  case  rn  —  2,  the  frequencies  Gq(u,  v),  . . . ,  G3(u,  v )  are  given  by  linear  combinations 
of  only  four  frequencies  F(u,  v),  u  G  {u,  u},v  G  {v,  v\  up  to  the  uncertainty  Z: 


G0(u,v) 

Hq(u,v)  H0(u,v )  H0(u,v)  H0(ii,v) 

F(u ,  v) 

Z0(u,v) 

Gi(u,v) 

H\{u,v)  Hi(u,v)  H\(u,v)  H\(u,v) 

F(u ,  v) 

-U 

Zi(u,v) 

G2(u,v ) 

H2(u,v)  H2(u,v)  H2(u,  v)  H2(u,v ) 

F(u ,  v) 

1 

to 

9" 

G3(u,v) 

H3(u,v)  H3(u,v)  H3(u,v)  H3(u,v) 

F(u ,  v) 

Zs(u,v) 

Hence  the  matrix  A  is  block  diagonal  up  to  a  permutation,  with  blocks  corresponding  to  AUjV,  - J  <  u,  v  <  f. 
It  follows  that  |  A  |  =  n„ „  Since  the  Fourier  transform  preserves  the  determinant  magnitude,  |z4|  = 

1^1  =  EL»  \A*,V  I-  ■ 


To  analyze  the  influence  of  motion  blur,  we  factor  the  terms  in  \AU^V\: 

H.j(a,  b )  =  0(a ,  b)C(a,  b)Mj(a,  b)Dj(a,  b ) 

with  a  G  {u,  u},  b  G  {v,  v}.  0{a,  b )  is  the  Fourier  transform  of  the  optical  transfer  function,  C(a ,  b)  is  the 
transform  of  the  detector’s  integration  function,  Mj(a,  b )  is  the  transform  of  the  motion  blur  point  spread 
function,  and  Dfa,  b )  is  the  transform  of  the  sensor  displacements  S(x  —  xj,  y  —  yf). 

Let  {lj}j= o  be  the  vectors  describing  the  motion  blur  path,  so  that  during  integration  the  projected  image 
Qj  moves  at  a  constant  velocity  from  —  ^  to  ^  (measured  in  the  high  resolution  coordinate  system).  The 
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transform  of  the  motion  blur  is  given  by: 


^  sin(mt^w) 

Mj(a,  b )  =  sinc(mlj  w)  = - 

7:mljw 

with  w  =  [a,  b\T . 

Let  {xj,yj}j= o  be  the  displacements  of  the  input  images  {gj}|=0,  respectively,  with  x0  =  0,yo  =  0.  The 
Fourier  Transform  Dj(a ,  b )  of  the  displacements  8(x  —  Xj,  y  —  y:i )  is  given  by: 

2  ni(axA-\-byA)  2n  i(ux  A-\-vy  a)  2ni((a  —  u)xa  +  (6  —  v)yA  ) 

7-^,  /  7  \ - 1 - 1 - - 2 - 2_ 

Dj\a,b)  =  e  n  =  e  *  e 

Dj(a ,  6)  is  expressed  as  a  product  of  two  terms.  The  first  term  is  common  to  all  pairs  (o,  b ),  and  hence  can  be 
factored  out  of  the  determinant.  Similarly,  the  terms  0(a,  6),  C(a ,  b )  are  common  to  all  images,  and  can  be 
factored  out  of  the  determinant.  It  follows  that: 


where 


Buv 
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M1(u,v)e~ns[ 

'A*i  M\(u, 

v)e~ 

-iirs(v)yi 

Mx(u, 

^e-i7v{s{u)xi+s{v)y1) 

M2(u , 

v ) 

M2(u,  v)e~'lnsi 

m2 (U; 

v)e~ 

-iTTs(y)y2 

M2(u, 

fi'je-i'K(s(u)x2+s{y)y2) 

M3(u, 

v) 

M3(u,v)e~i7TS> 

'A*3  M3(u, 

v)e~ 

-iirs(v)y3 

M3(u, 

fi'je-i'K(s(u)x3+s(v)y3) 

(6) 


(7) 


and  s(u)  is  an  abbreviation  for  the  sign  function  s(u)  =  sigri(u). 

The  influence  of  motion  blur  on  the  volume  of  solutions  is  therefore  expressed  in  the  matrices  Buv.  Since  the 


volume  of  solutions  vol(A)  = 


\A\ 


depends  on  the  image  size,  we  define 


S(A)  = 

so  that,  according  to  proposition  1  and  equation  6, 

vol(A)  = 


ni* 


(8) 


-1 

/  \  -1 

JJ  \Au,v\ 

oc 

nw 

\  u,v  / 

=  s(A)"  • 


(9) 


To  conclude,  s(A)  is  a  relative  measure  for  the  volume  of  solutions  that  is  independent  of  the  optical  blur 
and  detector’s  integration  function,  and  is  normalized  to  account  for  the  image  size.  The  generalization  of 
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the  above  results  for  an  arbitrary  integer  magnification  factor  m  is  straightforward,  and  is  omitted  in  order  to 
simplify  notations. 

The  lower  bound  for  s(A)  was  derived  using  the  following  inequality  for  a  A :  x  k  matrix  P  [14]: 


IPI  < 


l|F|| 


k 

2  \  2 


(10) 


with  ||  •  \\p  as  the  Frobenius  norm.  In  order  to  bound  s(A)  we  define  a  block-diagonal  matrix  B  of  size  n2  x  n2 
with  B(mu-m+j,mv-m+k)=Buv(j,k).  Using  inequality  10  on  equation  8: 


s(T)  — 


\B\-^ 


< 


\B\\ 


2  \  -2 


(11) 


The  matrix  B  has  m2n2  non-zero  values,  each  of  the  form  eixsim:(rnljwi:)  for  some  x.  The  Frobenius  norm 
of  B  is  hence 

m2  —  1 

H-biIf  =  ^2  sinc2(m^w),  (i2) 

j= o  wecxc 

with  C  =  {— \  +  ^}fc=o-  As  n  goes  to  infinity,  the  sums  are  replaced  by  integrals: 


m  —  1 


lim  -2\B\\l  = 

n— ►  oo  Tl 


/  sine2  (mB  w) . 


(13) 


The  integrals  were  solved  using  a  symbolic  math  software.  For  a  given  line  magnitude  ||L||,  the  maximal 


values  of  the  integrals  are  obtained  when  lj  is  oriented  by  45  degrees.  The  lower  bound,  appearing  in  Figure  2, 

rlT 

and  for 


is  therefore  the  value  of  equation  1 1  using  equation  13  for  a  45  degrees  oriented  blur  l  = 
a  magnification  factor  m  —  2: 


\/2  V2 
2  ’  2 


s(A)  <  4 


sine2  (mlf  w) 


'we  —  x  — 


Appendix  B.  Optimal  Spatial  Displacements 

We  show  that  when  there  is  no  motion  blur  (or  the  motion  blur  is  common  to  all  images),  the  four  grid 
displacements  {(0, 0)  (1, 0)  (0, 1)  (1, 1)}  (in  the  high  resolution  coordinate  system)  are  optimal  for  super  res¬ 
olution  in  terms  of  the  volume  of  solutions.  A  similar  result  was  shown  in  [10]  measuring  the  super  resolution 
quality  using  perturbation  theory. 


Proposition  2  Consider  the  imaging  process  as  defined  in  equation  4.  Assume  the  filters  {h/.  }^=0  have  the 
same  spatial  blur,  yet  different  displacements  {xk,  Vk}k= o>  Le-  hfc  =  h  *  S  (x  —  Xk,  y  —  Vk)  for  some  filter  h. 
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Then  vol(A)  ( equation  2)  is  minimal  for  displacements  {(0,  0)  (1, 0)  (0, 1)  (1, 1)}  in  the  coordinate  system  of 
the  high  resolution  image. 

Proof:  Let  H  be  the  Fourier  transform  of  h.  From  Proposition  1  and  equations  6, 7,  it  is  sufficient  to  prove 
the  maximality  of\BUtV  \  for  all  frequencies  (u,v).  In  this  case,  since  the  images  share  the  same  spatial  blur, 
the  motion  blur  can  be  folded  into  H,  and  equation  7  simplifies  to: 

111  1 

2  e~i7Ts(u)xi  e-i7Ts(v)yi  e-i7r(s(u)xi+s(v)yi) 

\  e-ins(u)x2  e-iFs(v)y2  e-i7r(s(u)x2+s(v)y2) 

\  e-ins(u)x3  e-ins(v)y3  e-in(s(u)x3+s(v)y3) 

The  rows  of  Bu,v  have  the  same  norm  for  all  assignments  of  yif)}-  Hence ,  the  determinant  is  maximized 
when  the  rows  are  orthogonal.  The  rows  are  orthogonal  if  and  only  if 

V/c,  /,  (1  +  e^is{u){xl~xk)  _|_  e7tis(v)(yi-yk)  _|_  g7 xis{u){xi~xk)  e^is{v)(yi~yk)^  _  q 

=>  Vk,  l,  (1  +  e’rts(«)(*i-*fc))(  1  4-  e™(t>)(w-yfc))  =  o 

which  is  satisfied  when  for  every  k,  l  either  \xi  —  xf\  =  1  or  \yi  —  y^\  =  1.  This  condition  is  satisfied  by  the 
above  displacements  {(0,  0)  (1,  0)  (0, 1)  (1, 1)}  ■ 

Note  that  there  are  other  displacements  that  maximize  \A\,  for  example  (0, 0)  (1, 0)  (x,  1)  ( x  +  1, 1)  for  any 
x  G  TZ. 
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